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RATIONAL GENERATING SERIES EOR AEEINE PERMUTATION PATTERN 

AVOIDANCE 

BRANT JONES 


Abstract. We consider the set of affine permutations that avoid a fixed permutation pattern. Crites has 
given a simple characterization for when this set is infinite. We find the generating series for this set using 
the Coxeter length statistic and prove that it can always be represented as a rational function. We also give 
a characterization of the patterns for which the coefficients of the generating series are periodic. The proofs 
exploit a new polyhedral encoding for the affine symmetric group. 


1. Introduction 

The affine symmetric group Sn is an infinite group that arises naturally in various geometric, combi¬ 
natorial, and algebraic contexts. In this work, we are concerned with the enumeration of various subsets 
of this group. Since the group is infinite, we consider “refined” counts of elements based on the Coxeter 
length statistic £{w) that describes the minimal number of generators needed to factor m G 5,^ in a certain 
standard group presentation of Sn- We will use the language of generating series to describe our results: 
For a given subset S C Sn-, we form the series Ylwes using a formal variable x and attempt to find 
a closed form for this expression. The associated enumerating sequence is the sequence of coefficients 
which counts the number of elements of each given length. These are related; for example, the enumer¬ 
ating sequence is given by a linear constant-coefficient recurrence precisely when the generating series 
can be expressed as a rational function. 

One of the first results in this direction is due to Bott IIBot56l who gave a general method to compute 
the Poincare series that describes the Betti numbers for the associated compact Lie group. Combinatori- 
ally, this is the generating series by Coxeter length for the entire group S = 

Theorem 1.1. llBot56l We have 

Uw) {I + x){l + X + X^){1 + X + + x'^) ■ ■ ■ {I + X + + x'^ ^ - 

(I-.-.) - 

weSn 

Although his motivation and proof were topological, it is relatively straightforward to give a com¬ 
binatorial proof by induction using the (so-called “parabolic”) subgroups obtained from subsets of the 
standard generators (see llHum901 (5.12)]). These subgroups turn out to be finite symmetric groups, for 
which the generating series is given by the numerator in Theorem 11.11 We give a new combinatorial 
proof for the denominator in Bott’s formula in Corollary 12.21 

Recently, Crites gave a natural extension of permutation pattern avoidance for the affine symmetric 
group as part of his thesis work with Sara Billey IIBC12I to characterize the rationally smooth Schubert 
varieties of affine type A. In HCrilOH . he also enumerated the number of affine permutations avoiding 
various fixed patterns, and proved the following remarkable structure theorem. 

Theorem 1.2. BCrilOll Let p be a finite permutation and n > 2. There exist only finitely many affine 
permutations of size n that avoid p if and only if p avoids the classical permutation pattern [321]. 
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Even when there are infinitely many affine permutations of size n that avoid a fixed pattern p, we can 
still consider the length generating series 

W&Sn 
w avoids p 

Such series first appeared in Hanusa and Jones’ IIHJIOI enumeration of the [321]-avoiding affine permu¬ 
tations. It is shown there that the coefficients of the length generating series for p = [321] are periodic. 
These [321]-avoiding affine permutations are also known as the fully commutative elements of affine type 
A. More recently, Biagioli, Jouhet, and Nadeau IIBJN13I have described the length generating series for 
fully commutative elements in other affine types, and they turn out to be periodic there as well. In fact, 
they propose the problem of determining which Coxeter groups have a periodic generating series asso¬ 
ciated to their subset of fully commutative elements. This would generalize Stembridge’s classification 
listed . 

In this work, we consider the dual problem of classifying the periodic patterns within the affine sym¬ 
metric group. While any generating series with periodic coefficients can be expressed as a reduced 
rational function with denominator 1 — x‘^, it is not obvious that the Fp^n{x) series are even rational in 
general. One standard way to show that a counting problem is solved by a rational generating series is 
to produce a bijection to directed paths in a finite graph (or equivalently, words in a regular language). 
Stanley IISta97ll refers to this as the “transfer matrix method.” In fact. Brink and Hewlett have described 
a clever finite state automaton that recognizes a canonical reduced expression for each element of a fixed 
Coxeter group (see IIBH931 or IIBB051 Chapter 4]); Casselman has also contributed significantly to make 
their ideas practical for efficient implementation in software (see IICas95L for example). We initially 
attempted to modify these constructions to filter the affine permutations based on pattern avoidance cri¬ 
teria. At this stage, however, it appears that pattern avoidance is not sufficiently related to the group 
structure for this approach to work in general. 

Recently, we have turned instead to a set of ideas based on geometric convexity. Consider a rational 
polyhedron P defined as the set of solutions in M"' to a set of linear inequalities with integral coefficients, 
and suppose that we would like to count the lattice points in Z” n P. To be more general, we consider 

the encoding series 


Pp{xi,...,Xn) ■■= ^ 

for these points in the formal variables xi,, Xn. Brion’s formula (see IIBR07II or llBar021l i states that 
this encoding series is simply the sum of the encoding series for each of the “tangent cones” formed 
by the rays emanating from a vertex of P. Moreover, it is straightforward to see (after using inclusion- 
exclusion if the cones are not simple) that the encoding series for these tangent cones are all rational, and 
so any generating series obtained by specializing the Xi will be rational also. 

More precisely, we show in Section [2] how to coordinatize (the minimal length coset representatives 
of) Sn as the set of lattice points (zj,..., Zn-i) in the nonnegative orthant with Coxeter length 
given by ~ Enumerating these points recovers the denominator of Bott’s formula. 

However, it turns out that the subset of lattice points corresponding to the p-avoiding affine permuta¬ 
tions, for a fixed pattern p, is not necessarily convex; see EigureOb). We then show that it is possible to 
decompose into a disjoint union of (n — 1)! shifted, dilated cones, each of the form 


{(^1) 2f2, . . . , (n — l)tn-l) -h (& 1 , . . . , bn-l) '. U € Z>o}. 




















RATIONAL GENERATING SERIES FOR AFFINE PERMUTATION PATTERN AVOIDANCE 


3 


If we restrict to each C^, then we can prove that the p-avoiding affine permutations do form a polyhedral 
set. In fact, we give explicit defining inequalifies fhaf include some addifional coordinafes for conve¬ 
nience, and fhen projecf fo fhe f-coordinafes fhaf parameferize each C^. Af fhe end of fhis process, we 
can apply Brion’s formula fo compufe fhe enumerafing series and conclude fhaf if is rafional. 

Lef us pause fo mention fhaf fhis consfrucfion seems likely fo be useful in ofher confexls. For example, 
the Coxeter hyperplane arrangement of affine fype A„_i in M” is given by Xi — Xj = k for 1 < i < 
j < n and A: G Z. The complemenf of fhese hyperplanes in M” is a collecfion of regions. If fums ouf 
fhaf fhese regions are in bijecfion wifh affine permufafions, and so enumerafing fhese regions using a 
sfafisfic defined by counfing fhe number of hyperplanes fhaf separafe a region from a fixed region af fhe 
origin resulfs in fhe same generating series as Boff’s formula. There is some recenf inferesf IIArml31 
IFVIOI in sfafisfics and generating series for regions of fhe exfended Shi arrangemenfs (which are a 
subarrangemenfs of fhis one), and affine paffem avoidance may be a useful fool for refining fhis geomefric 
picfure. 

The generafing series we have been considering also arise in cerfain lattice pafh enumerafion problems; 
see llBDLPPOll IBDLFP981IBJN13I . In facl, fhe enumerafion for p = [321] in IlHJlOl used a recursive 
fechnique of Bousquef-Melou IIBM961I developed for fhis confexf involving (^-Bessel functions fhaf, while 
powerful, leaves fhe generafing series in a form fhaf is somewhaf opaque. Our decomposifion of fhe 
coordinafe space for fhese objecfs info shiffed dilafed cones seems likely fo offer some new insighfs info 
fhese fypes of recursive sysfems. 

Once we know fhaf our Fp^nix) generafing series are rafional, fhere are fhree possibilities for fhe 
sequence of coefficienls: fhey musf be evenfually zero, evenfually repeal, or are unbounded. The firsl 
case is characferized by Criles’ Iheorem, and in Secfion[3l we begin fo characlerize fhe periodic patterns. 
We are aided by fhe facl fhaf if suffices fo characlerize fhe periodic patterns in a single space, wifh 
n = 3. Slaled in terms of classical permutation patterns, our result essentially requires p to avoid an 
infinite family of patterns from St,Ss, Siq, S 12 , S 14 ,...; see Figure [5] and Theorem 13.1 II 

When p cannot be embedded into any element of Sn then the generating series Fp^n{x) is simply 
given by Bolt’s formula, which is not periodic (unless n = 2). It remains an open problem to give a 
characterization in terms of p for when this occurs. We do not address this here although there are stan¬ 
dard techniques from convex geometry that can be applied to the polyhedra we define for any particular 
pattern of inferesf. 

There are many open direclions in fhis area, for bolh undergraduafe and professional researchers. 
Almosl any of fhe classical problems associafed wifh permulalion patterns, such as classificalion of Wilf 
equivalence classes, pattern packing, or asymplofic behavior, could be posed in fhe affine setting; see 
IIB6nl2ll for an infroduclion fo fhese classical resulfs. If would also be inferesfing fo exfend our geomefric 
framework fo sfudy bivariafe generafing series of fhe form YIwgScs x^^'^'^y'^. Moreover, modifying fhe 
geomefric framework fo handle multiple patterns would allow us fo sfudy fhe {[3412], [4231]} class from 
IIBC12I in defail. 


2. Polyhedral Structure 

2.1. A polyhedral encoding of the affine symmetric group. An affine permutation of size n is a 

bijecfion m : Z ^ Z satisfying w{i + n) = w{i) + n for all z G Z, and m(l) -|- w{2) w{n) = 

1-I-2-I-- ■ -Fn. We refer fo fhe (infinife) image sequence (..., r(;(—2), ?/;(—1), r(;(0), m(l),...) of re as ifs 
Z-notation. By fhe firsl properly, we can complelely specify an affine permulalion by ils base-window 
[r(;(l), r(;(2),... , r(;(n)]. When we do fhis, fhe Z-nolalion is obfained by decomposing fhe image info 
windows of size n, where fhe zlh window conlains fhe enlries of fhe base window wifh each value in fhe 
window shiffed by in. (In fhis paper, we denofe window boundaries wifh a j symbol.) 
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The affine symmetric group Sn consists of all the affine permutations of size n, with composition 
of functions as the group operation. It follows directly from the definitions that [wi,W 2 , ■ ■ ■, Wn] is the 
base-window notation for an affine permutation if and only if Wi = and the residues {wi 

mod n) are all distinct. 

As a group, Sn is generated by the n adjacent transpositions of entries in the Z-notation (where each 
transposition acts on all windows simultaneously). The minimal number of such transpositions into 
which w can be factored is known as the Coxeter length of w, denoted l{w). 

Given a permutation p ^ and an affine permutation w G Sn, we say that w contains the pattern 
p if there exist positions h < i 2 < ■ ■ ■ < ik whose Z-notation values w{ii),w{i 2 ), ■ ■ ■, w{ik) are in the 
same relative order sls p\,p 2 , ■ ■ ■ ,Pk- Note that these positions need not be restricted to the base window. 

When the entries in the base-window notation for w are sorted increasingly, we call w a minimal 
length coset representative. (See IIBB051 IHumQOI for motivation and details.) We will denote the 
subset of minimal length coset representatives by 5° C Sn- Then each w ^ S^ corresponds to an 
abacus diagram as follows. Begin with an array having n columns and countably many rows. Label 
the entry in the fth row and jth column of the array by the integer j -|- ni, where 1 < j < n. In figures, 
we will draw the rows increasingly up the page, and columns increasingly from left to right. Then these 
labels linearly order the entries of the array, which we refer to as reading order. We call the entries 
{1 + kn,2 + kn,... ,n + kn} the fcth level of the array. To create our diagram, we highlight certain 
entries in the array; such entries are called heads and will be circled in figures. Entries that are not 
beads will be called gaps. To encode w, we let the entries in the array corresponding to the base-window 
notation for w be beads, and we refer to these as the defining beads. To complete the diagram, we create 
beads at all of the entries below each defining bead, lying in the same column. All of the other entries in 
the diagram are gaps. We call this completed diagram the abacus diagram for w. 



18 

12 

6 

0 

-6 





Figure 1. An abacus diagram for w = [—12, —8, 2,9,13,17] with w = (4,10, 7,4,4) 
and w = (0, 3,3, 2,3). 

Observe that the defining conditions on the base-window notation imply that the levels of the defining 
beads in an abacus diagram must sum to zero. We refer to this by saying that the abacus diagram must 
be balanced. Hence, the base window notation includes a redundant coordinate. To remedy this, we 
can represent any minimal length coset representative w by its gap vector w = (rui,..., Wn-i) where 
Wi records the number of gaps between the i and (f -|- l)st defining beads in the abacus for w, ordered 
increasingly. Alternatively, we may specify w by its delta vector 

W = {W2- Wi,W^ - W2, - - - ,Wn - Wn-l)- 
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This vector records the number of entries (which may be beads or gaps) in the abacus diagram between 
each successive pair of defining beads. 

Observe that any nonnegative integer vector is the gap vector for a unique abacus diagram. To see this, 
simply place the largest defining bead arbifrarily on fhe array, and fhen place each of fhe smaller defining 
beads wifh consecutive disfances as prescribed by fhe given gap vector. To balance fhe abacus, subfracf 
the sum of the levels of the defining beads from fhe position of each defining bead. The resulf will be fhe 
unique balanced abacus having fhe prescribed gap disfances befween consecufive defining beads. 

Proposition 2.1, The Coxeter length ofw is given by i{w) = Wn-i + 2wn-2 + 3tPn-3 + • • • + (n — l)tt;i. 


Proof. This is a “folklore” resulf fhaf is somefimes sfafed in a slighfly differenf form: To compufe fhe 
Coxefer lengfh of fhe elemenf encoded by an abacus diagram, counf fhe number of pairs (6, g) where b 
is a defining bead and p is a gap fhaf preceeds b in reading order. 

Once we franslafe fhe action of Sn to the abacus, it is straightforward to prove this result by induction 
on £{w ); simply check that each length increasing adjacent transposition adds a single new (6, g) pair. □ 

As a corollary to this development, we may view the (gap vectors of) minimal length coset representa¬ 
tives as lattice points in the nonnegative orthant, which is a prototype for our polyhedral encoding. When 
we enumerate these points with respect to the Coxeter length statistic, we recover the classical result of 
Bott for type A. This seems to be a new proof, and it is an open problem to give analogous proofs for the 
other affine Weyl groups. 


Corollary 2.2. (Bott) We have 




v£S° 


(l-g)(l-q2)...(l_gn-i)- 


Proof. By the development above, the encoding series for the gap vectors is 

7 Jb ^ Jb ry 


9n-l — 
•^n-1 — 




(1 - Xi)(l - X 2 ) • • • (1 - Xn-l) ' 


>0 


By Proposition 12.11 we can then obtain the length generating series by substituting g” * for Xj. This 
yields the result. □ 


To uncover the polyhedral structure that will be useful in conjunction with patterns, we need a further 
refinement. We say that an abacus on n columns is minimal if its delta vector uses only entries between 
1 and n — 1. For example, the minimal abaci in n = 4 are shown below in Figure |2] 


Proposition 2.3. There are (n — 1)! distinct minimal abaci on n columns. 


Proof. We argue by induction, the result being clear if n = 2. Assume the formula holds for abaci on 
n — 1 columns. To form a minimal abacus on n columns, we can start with a minimal abacus on n — 1 
columns, insert a new column containing a new largest defining bead in any of n — 1 distinct positions, 
and rebalance the resulting n column abacus. Moreover, every minimal abacus on n columns arises this 
way. Hence, the formula holds by induction. □ 

Given w G S'”, we can project tu to a minimal abacus by repeatedly removing multiples of n entries 
between consecutive defining beads and then rebalancing the diagram. We call the minimal abacus 
obtained in this way the bias of w. Equivalently, the bias 6 of m is specified by its delta vector b = (u)i 
mod n,W 2 mod n, ... ,u)n-i mod n). Let BIASn, denote the set of (n — 1)! possible biases on n 
columns. 


Example 2.4. The bias of the abacus shown in Figure IZTl is given by 6 = (4,4,1,4,4). 
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Figure 2. The (n — 1)! different minimal abaci in n = 4 


Lemma 2.5. 'We can decompose the set of gap vectors into a disjoint union of shifted dilated cones 

S°n = K-^= U C 

feGBIASn 

where 

{(^1) 2^2, ■ ■ ■ ,(n — + (6l, . . . , bn-l) ■ ti G Z>o}. 

Note that each element of C]j is shifted by the same vector {bi,, bn-i) that depends only on b. 
Hence we will refer to points in CJj by their f-coordinates, (ab)using the notation (ti,..., tn-i) G CJj. 

Proof. We claim that each gap vector w = {wi, ..., Wn-i) G exists in precisely one of the Cjj 

sets. To see this, draw the abacus associated to the point w. Suppose the i and {i + l)st defining beads 
have more than n entries between them. Then we delete one entire level of the array between them (and 
then renumber the remaining entries of the array). This maneuver removes i gaps from the coordinate 
Wi (since there will he n — i beads on the level we remove), which is equivalent to removing 1 from 
coordinate f. Repeat this process until every consecutive pair of defining beads is separafed by less fhan 
n enfries, and fhen rebalance fhe abacus (by subfracfing fhe sum of fhe levels of fhe defining beads from 
fhe posifion of each defining bead). By definifion, fhe resulf will be one of fhe minimal abaci. 

Moreover fhis process is reversible since we can recover w by sfarfing wifh fhe minimal abacus, 
inserting levels as prescribed by fhe f coordinafes, and rebalancing. Hence, fhe poinf of CJj is unique. □ 

Until now, we have focused on fhe minimal lengfh cosef represenfafives S°. From fhe lengfh-addifive 
parabolic decomposifion in fhe fheory of Coxefer groups, we have fhaf fhe base-window nofafion of each 
w € Sn can be decomposed info a sef of values fogefher wifh a “sorfing permufafion” v G Sn- The sef 
of values is represenfed by some u G S°, and we have seen fhaf fhese furfher decompose info subsefs of 
elemenfs having fhe same bias. The finife permufafion v is fhe unique finife permufafion having enfries in 
fhe same relative order as fhe base-window nofafion for w. We call v fhe flattening of w, and it follows 
that i{w) = £{u) -h £{v). 

Hence, we can extend our polyhedral embedding to Sn by simply taking n! copies of the embedding 
for S°. Thus we let denote the set of w € Sn whose bias is b and whose flattening is equal to v. 
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Corollary 2.6. 'We have the disjoint union 

Sn= \J 4 -,. 

beBIASn 

V&Sn 

Example 2.7. In n = 3, we can draw the minimal length coset representatives as a set of lattice points 
in the plane. There are two minimal abaci, given by d = (1,1) and b = (2, 2) with offsets given by 
a = (0,0) and b = (0,1), respectively. Then the gap vectors are a disjoint union of two 

cones, where the second coordinate has been dilated by 2 and the cones have been shifted by (0,0) and 
(0,1), respectively. 


The entire affine symmetric group S 3 consists of six copies of this set of lattice points, one for each 
choice of flattening. 

2.2. Patterns. Fix a permutation pattern p € Sk, together with a bias b and flattening v. We will first 
explain how to characterize the elements of that contain an instance of p. Recall that an instance of 
the pattern p in the affine permutation m is a choice of k positions in the Z-notation for w whose values 
have the same relative order as p. To coordinatize this, we consider two pieces of data associated to an 
instance: a strand assignment, and a window assignment. 

Definition 2.8. Let the strand assignment of an instance be the function vr assigning each entry of p 
to an entry of the base window of w, where 7r(z) = j means that pi is represented by some positional 
translation of the jth largest value of the base window (where j = n represents the largest value). 

The set of potential strand assignments for p is finite, consisting of all sequences of length k with 
values from n}. In fact, it is not difficult to discover a further requirement for strand assignments. 

Lemma 2.9. Let ir be a strand assignment for p. Then either every inversion in p must correspond to a 
strict inversion in vr, or else the set of w containing p with strand assignment vr is empty. 

Proof Suppose i < j and pi > pj. If 7r(z) < 7r(j) then the elements representing pi and pj would 
necessarily be increasing in w, a contradiction. □ 

Definition 2.10. The window assignment of an instance is the vector (ci,... ,Cfc_i) where Cj is the 
number of positional window boundaries between the entries representing pi and pj+i in w. If pi and 
Pi+i lie within the same window then we set Ci = 0. 

Example 2.11. Consider the highlighted instance ofp= [24351] in the Z-notation for w = [—9,4,11]: 

(-15, - 2,5| - 12,1,8| - 9,4,11] - 6, 7, 14|-3,10,17| • • •) 

Then, vr = [2,3, 2,2,1] and c = (0,1, 2,1). 
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It is clear that the strand assignment and the window assignment completely determine a pattern 
instance in w. Given an affine permutation w in Z-notation, we can recover the f-coordinates from as 
follows. 

Lemma 2.12. Given w £ S° let j >0 be maximal such that w{i + 1) > w{i + jn). Then, ti equals the 
number of window boundaries lying between w{i + 1) and w{i + jn). 

For Example 12.111 we find t 2 = 2 since there are two window boundaries lying between entries 11 
and 10. Similarly, ti = 4. 

Proof. Work by induction starting from a minimal abacus. In a minimal abacus, there are no adjacent 
inversions between windows in Z-notation and all the ti are zero. Each time we add one to a ti coordinate, 
we adjust w by adding n to each of the n — i largest values in the base window, and then subtracting re — f 
from each of the values in the base window (to rebalance). This places one new window boundary that 
is counted by the description in the statement, and preserves all of the other window boundaries. □ 

We next characterize the f-coordinates of points in that contain an instance of p with strand 
assignment tt. To accomplish this, we highlight some data in (p, vr). 

Definition 2.13. Given a pattern p £ and a strand assignment vr for p, we say that an upshift is a pair 
j < i such that pj = pi + 1 and 7r{j) > 7r(v). A downshift is a pair i < j such that pj = pi + I and 

Tr{i) > 7r{j). 

Example 2.14. Consider (p, vr) = ([24351], [2, 3, 2, 2, Ij). The values 1 and 2 in p form an upshift that 
we denote (positionally) as (1 < 5). Values 2 and 3 are both assigned to strand 2, so no shift takes place. 
Values 3 and 4 form the upshift (2 < 3), and values 4 and 5 form the downshift (2 < 4). 

We are now in a position to prove our main result in this section. 

Theorem 2.15. For each (p, vr), the set 

{t = {ti,..., tn-i) £ '■ t contains an instance of p using strand assignment vr} 

consists of the integer points in a rational polyhedron. 

Proof. We first prove that the set 

{{t, C) £ Cjjy xZ>Q^ : t contains an instance of p using strand assignment vr and window assignment c} 

consists of integer points in a rational polyhedron. Then, we can project onto the i-coordinates to obtain 
the result. More precisely, we will give a collection of integral linear inequalities that the (f, c) coordi¬ 
nates satisfy exactly when they describe a valid instance of p in the affine permufafion corresponding fo 
t. Then fhe fundamenfal Minkowski-Weyl Theorem for convex polyhedra allows us fo describe fhis sef 
as fhe Minkowski sum of a (bounded) polyfope and a recession cone of rays. To perform fhe projection, 
we simply ignore fhe c coordinafes in fhis laffer description. (See IIBR07I or llZie951 Eecture 1] for an 
infroducfion fo fhese ideas.) 

Given a window assignmenf c, we imagine placing fhe values of p info an affine permufafion w (whose 
values are fo be defermined) in increasing order. Whenever we place a larger value in a position fo fhe 
righf, on fhe same sfrand or higher, we impose no condifions on fhe ti because fhe sfrands are necessarily 
increasing in w. However, if we place a larger value fo fhe leff, we musf increase fhe sfrand and so fhis 
pair of consecufive values is an upshiff. Also, if we place a larger value fo fhe righf on a lower sfrand, 
fhis pair of consecufive values is a downshiff. These do impose condifions on fhe ti. 

By Eemma 12.121 we have fhaf ti represenfs fhe maximal number of posifional window boundaries 
lying befween (franslafes of) fhe Ah and (v -|- l)sl largesl elemenfs of fhe base window fhaf can form an 
inverted pair. Similarly, ti + U^i + ■ ■ ■ tj-i + [hj] represenfs fhe maximal number of posifional window 
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boundaries lying between translates of the ith and jth largest elements of the base window that form an 
inverted pair. Here, [bij\ = hb(j-i) j jj^g (constant) contribution from the bias. 

Therefore, when we have an upshift from strand 7r(i) to strand 7r(j), we must ensure that + 
• • • + i 7 r(j)-i + L^ 7 r(i), 7 r(j)J l^Tgc cuough to cusurc that the entries at positions j and i are 
inverted. These positions are separated by cj + Cj+i + • • • + Ci_i window boundaries. 

Hence, for each upshift {j < i) we include an inequality of the form 

+ ^7r(j) + l + ■ ■ ■ + i7r(j)-l + L^7r(i),7r(j)J > Cj + Cj + 1 + ' ' ' + Cj_l. 

Using similar reasoning for each downshift (z < j), we include an inequality of the form 

^7r(j) + ^7r(j) + l +-h ^7r(i)-l + L^7r(j),7r(i)J < Ci + Cj+l H-h Cj-l — 1. 

The choice of flattening enters in the initial conditions on q, which specify the minimal number of 
window boundaries between the i and {i + l)st position in the pattern instance. Since these positions 
must be increasing, we include 


a > 



if n ^(7r(z)) < n ^(7r(z + l)) 
otherwise 


for each 1 < i < k. The initial conditions U > 0 should also be included. 

Our strategy to place p into the affine permutation will succeed if all of these linear inequalities are 
satisfied. If any are not satisfied, then we will have a pair of consecutive values from p whose represen¬ 
tatives in the affine permutation do not faithfully represent the pattern. Hence, the integer points of this 
polyhedron form precisely the set given in the beginning of the argument. After projection, we obtain 
the result. □ 


Example 2.16. The linear inequalities obtained for {p, tt) = ([24351], [2, 3,2, 2,1]) with b = (0,0) and 
V = [123] are: 

ft > Cl + C 2 + C 3 + C 4 , t2 > C2, ^2 < C 2 + C 3 - 1, 

Cl > 0, C 2 > 1,C3 > 1,C4 > 1; ti > 0,t2 > 0. 

Definition 2.17. We will refer to the rational polyhedron constructed in the proof of Theorem 12.151 by 

It turns out that the bias and flattening parameters do not change the polyhedra very much. 

Lemma 2.18. Let bo be the bias given by b = (0,0,... , 0) and vq be the identity permutation in Sn- 
Then for any other choice ofb^ BIAS^ and v € Sn, we have that CJ^^{p, tt) has the same set of infinite 
rays as 

As a result, we often drop b and v from our notation, and let C"‘{p, tt) = tt). 

Proof Write the polyhedron Cfy{p, vr) as the solution set to a collection of linear inequalities. We can 
use a matrix A and multiplication by —1 to write this in a standard form Ax < b. It is well-known 
(and straightforward to verify) that changing b cannot change any of the infinite rays in the solution set. 
Since changing the bias or flattening parameters only alters the defining inequalities by a constant, and 
preserves all of the coefficients of the L and Ci, we obtain the result. □ 


Corollary 2.19. For any permutation pattern p and any n > 2, the generating series 

FpA^)= E 

W^Sn 
w avoids p 

is rational. Equivalently, the coefficient sequence is generated by a linear constant-coefficient recur¬ 


rence. 
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Proof. Using Brion’s formula (see IBR071 or IIBar021l ) together with inclusion-exclusion applied to The- 
orem l2.l51 we can obtain a rational encoding series for the points of each |J^ tt). The subsequent 

union of these sets over all b and v are disjoint, so we can simply add the encoding series together. Then, 
we specialize the encoding series by setting U to * for each 1 < i < n. The first exponent dilates 

the lattice to recover the gap coordinates as in Lemma IZ5l and the second exponent comes from Propo¬ 
sition [ZT] Finally, we subtract the result from Bott’s formula (which itself is rational) to enumerate the 
p-avoiding elements. □ 

Let us turn to some examples in n = 3 where we can draw pictures. 




FIGURE 3. (a) (p,7r) = ([321], [3,2,1]); (b) (p,7r) = ([2431], [3,3,2, Ij); 

(c) (p,7r) = ([24351],[2,3,2,2,l]). 


Example 2.20. In Figure|3]we have displayed some C^^(p, vr). In each of the examples, we have n = 3, 
V = [123], both biases are displayed superimposed, and (p, vr) vary. We have also drawn some of the 
hyperplanes of constant Coxeter length from which the contributions to the rational generating series can 
be computed. 

Observe that in Example (a) the counting sequence for the number of p-avoiding elements eventually 
stabilizes. In Example (b), we have a periodic sequence with period 2. Example (c) produces an un¬ 
bounded counting sequence (although other strand assignments provide a ray in the y direction that is 
missing for this assignment; the full counting sequence for this p turns out to be periodic). 

Warning 2.21. These polyhedra can be empty. Eor example, p = [7,1,0,4, 5,2, 8,10, 6, 9, 3] has only 
one strand assignment using 3 strands, and the corresponding C”(p, tt) polyhedron is empty. 

There are some natural questions about these polyhedra to which we do not currently know the answer. 

Question 2.22. If we fix the bias and flattening parameters, is the union 

TT 

over all strand assignments necessarily convex? (If so, this would dramatically simplify the computation 
of the encoding series.) 

Question 2.23. Given a pattern p & with j strands, we certainly need n > j in order to successfully 
embed p into Sn- By Waming \2.21\ this inequality is sometimes strict. Is there a simple way to describe 
the minimal size of an affine permutation that contains a given pattern p? 
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3. Periodic patterns 

Let ai denote the coefficients of the rational generating series Fp^n{x) from Corollary 12. 191 That is, a* 
counts the number of affine permutations of fixed size n and length i that avoid the fixed pattern p. Since 
the ai obey a recurrence, it follows that there are three possible types of behavior. 

Definition 3.1. We say that a permutation pattern p is finitely enumerated if the ai are eventually zero. 
We say p is periodic if the ai eventually satisfy ai = ai-N for some fixed N. Ofherwise, we say that p 

is unbounded. 

(To verify that this definition is etymologically sound, use the pigeonhole principle to show that when¬ 
ever tti is a bounded sequence that satisfies a recurrence using a fixed number of prior terms, then Oj is 
actually periodic.) 

Crites’ characterized the finitely enumerated patterns in IlCrilOII . and Hanusa-Jones gave the first 
example of a periodic pattern, p = [321], in IIHJIOII . Our goal in this section is to characterize all of the 
periodic patterns. 

Note that the classification in Definition 13.11 depends only on the denominator of the generating series 
and so the contributions from each bias and flattening must each fall into the same case by Lemma l2.18l 
For this reason, it suffices to work with the enumerating sequence for |J.^ vr) in this section. 

Definition 3.2. Given p € S'fc, let m be the length of the longest decreasing subsequence of p. In this 
situation, we say that p has m strands. 

We rephrase Crites’ Theorem from HCrilOll as follows. 

Theorem 3.3. (Crites) In each n, the permutation pattern p is finitely enumerated if and only if p has 
fewer than 3 strands. 

The following result then shows that periodic patterns can only exist on three strands. 

Proposition 3.4. In each n, if p has four or more strands then p is unbounded. 

Proof. Consider Bott’s formula for S'g. The sequence of coefficients is unbounded, and the affine per¬ 
mutations in all avoid p (since the length of the longest decreasing subsequence in any of them is 
clearly 3 or less). When n > 3, we can embed in € S'g into 5° by padding w with zeros on the left. This 
embedding is injective, the length of the longest decreasing subsequence in the image will be the same 
or smaller, and by Proposition 12. ll we do not change the Coxeter length. Hence, we obtain the result. □ 

Lemma 3.5. In each n, we have that p is periodic if and only if there exists a constant B such that 
IJ.^ C^{p, Tt) contains every point (ti,..., fn-i) that has two or more ti coordinates larger than B. 

Proof. Since the enumerating sequences are generated by a recurrence, we have that p is periodic if and 
only if there exists an upper bound for the values of the sequence. Also, the condition in the statement 
for the t coordinates is true if and only if it is true for the corresponding w gap vector coordinates. 

To prove the result, first suppose that at most one Wi coordinate (from the space of gap vectors) 
can become arbitrarily large when we avoid p. Then when we intersect with the hyperplane 

Wn-l + 2Wn-2 + 3Wn-3 H-h (n - l)u)i = i 

of points with fixed Coxefer length i, for large i, the unbounded coordinate actually becomes determined; 
it must take up the “slack” in this equation for all of the bounded coordinates. As a result, we only have 
a bounded number of p-avoiding gap vectors, so the sequence is periodic. 

Conversely, if there can be two unbounded gap vector coordinates when we avoid p, then one of these 
coordinates will be undetermined when we intersect with the hyperplane of fixed Coxeter length i, for 
sufficiently large i. Hence, the enumerating sequence is unbounded and so p is not periodic. □ 
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Corollary 3.6. In n = 3, we have that p is periodic if and only if\J^ vr) contains infinite rays in 
the ti and t 2 directions. 

We now turn to classify the periodic patterns in n = 3. Eventually we show that these are the only 
periodic patterns for any n. 

Definition 3.7. We say that (p, vr) is feasible if |J^ C^(p, vr) is nonempty. 

While it remains an open problem to provide a (simple) combinatorial characterization for feasibility, 
there are standard techniques from convex geometry (such as Fourier-Motzkin elimination or Lenstra’s 
algorithm for integer programming HSchShll ) that may be used to address this question. 

Definition 3.8. Let p € Sk with 3 strands, and let vr be a strand assignment for p. Consider the diagram of 
(p, vr) in which we represent pi by a point (i, pi) in the plane and label the point by its strand assignment 
7r(i). 

We say that two elements of the second strand pi and pj are linked below (above) if there exists an 
element of the first (third, respectively) strand lying below and right (above and left, respectively) of both 
of them. 

We say that two elements of the second strand are chained below (above) if there is a consecutive 
sequence of elements from the second strand between them that are linked below (above, respectively). 

A corner of (p, vr) consists of a triple (f, j, k) such that pi and pj are distinct elements of the second 
strand, and pk is an element of the first or third strand that lies inside the square having pi and pj as 
diagonal vertices. 

The comer is said to be tight if the elements pi and pj are chained below, or chained above. 

Some tight comers that are chained below are shown schematically in Figures |4] and [5] Points drawn 
in the same row or column can be resolved to a permutation by either perturbation of the points. Thus, 
each picture encodes several classical permutation patterns. 



Figure 4. Minimal tight corners 


Lemma 3.9. If (p, vr) is feasible and contains a tight comer then ti or t 2 is not a ray ofC^{p, vr). 

Proof Suppose (p, vr) has a tight corner. Without loss of generality, we may assume it is chained below 
as shown in the figures. Then, we claim that t 2 is not a ray. If it were, we could fix t\ and increase t 2 
arbitrarily. However, once ti is fixed, there is a maximum width for the strand 2 entries that are chained. 
Then we cannot increase t 2 past the distance limited by the strand 3 entry that is in the tight corner. □ 
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Figure 5. General form: tight corners 

Lemma 3.10. If {p, n) is feasible and does not contain a tight corner then both ti and t 2 are rays of 
C^{p,Tr). 

Proof We argue the contrapositive. Suppose, without loss of generality, that whenever ti is fixed there 
are only finitely many values for t 2 . If there were no corner of 2 entries enclosing a strand 3 entry, then 
we could separate strands 2 and 3, increasing t 2 arbitrarily. If the 2 entries defining fhe corner were nof 
chained, fhen we could slide fhem along fheir strand and thereby increase t 2 arbitrarily. Therefore, we 
must have a tight corner. □ 

Theorem 3.11. The pattern p is periodic in if and only if there exists a strand assignment vr that is 
feasible and does not contain a tight comer. 

Proof First suppose there exists such a strand assignment. Then, Lemma lS.lOl and Corollary 13. 6 1 imply 
the result. 

Next, suppose that no such strand assignment exists. If this is because no vr is feasible, then p is not 
periodic since the enumeration is given by Bott’s formula. 

So suppose that every feasible vr has a tight corner. We show that they all contain the same type of tight 
corner (i.e. are all chained above, or chained below). Fix some feasible vr and consider the “supporting 
entries” shown in light gray in the figures. If fhese enfries were nof presenf in p, if would be possible fo 
modify fhe strand assignment tt to get rid of the tight corner, a contradiction. 

Hence, the supporting entries must be present in p. But this implies that the strand assignments for 
the entries of the tight corner are forced in every strand assignment. Therefore, no tt' can contain the ray 
that is missing due to the tight corner of vr and LemmaThus, p is not periodic by Corollary 13.61 □ 

Finally, we complete the periodic classification for n > 3. 

Theorem 3.12. Let n > 3. If p is not periodic in then p is not periodic in S°. 
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Proof. Suppose for the sake of contradiction that p is not periodic in S'^, but p is periodic in S'”. Then, 
the values of the enumerating sequence for the p-avoiding elements of S° are bounded by some value B. 

We embed points (fi, ^ 2 ) from S 3 into S° by appending zeros on the left. After dilation, the length 
formula for these points is 


£{0, 0,... , 0 ,ti,t 2 ) = {n - l)t 2 + 2{n 
Hence, the points having fixed length k in S° satisfy 

1 


n — 1 


n — 1 


-k. 


2)ti. 


Since p is not periodic in S 3 , we have that |J^ C^{p, vr) does not contain rays in both the ti and £2 
directions. Since the points of |J^ C^(p, tt) must lie in the nonnegative quadrant, this implies that some 
ray with positive slope separates the p-containing lattice points from the p-avoiding lattice points in Z>g. 

Because the embedded points having fixed lengfh k in S° have negative rafional slope, fhere musf 
evenfually be some large value of k for which we obfain more fhan B disfincf poinfs (fi, f 2 ) such fhaf: 

(1) (0,..., 0, fi, f 2 ) has lengfh k in S°. 

(2) {ti,t 2 ) avoids p in S 3 . 

Since fhere are more fhan B of fhese embedded poinfs, some of fhem musf confain p. So suppose 
t = (0,..., 0, ti,t 2 ) confains p. Then fhere exisfs some sfrand assignmenf vr for which t is feasible in 
C”(p, tt). Nofe fhaf since fhe firsf n — 3 coordinates are zero, we cannof have any upshiffs involving 
sfrands {1, 2,..., n — 2}. Therefore, we can form a sfrand assignmenf vr' by changing all of fhese values 
fo n — 2, and fhe poinf t will still be feasible for C"'(p, vr'). Buf fhen fhe poinf t would also have been 
feasible for C^(p, tt"), where vr" is obfained from tt' by sending n — 2 fo 1, n — 1 fo 2 and n fo 3. This 
confradicfs (2) above. □ 


Theorem 3.13. Let n > 3. Ifp is periodic in S 3 then p is periodic in S°. 

Proof. By Theorem 13.111 fhere exisfs a feasible sfrand assignmenf tt whose solufions include bofh ti 
and t 2 as rays, so fhe solufions include all poinfs having bofh coordinafes larger fhan some B' . Lef 
t' = {ti,..., tn-i) € C"'(p, tt) wifh fwo coordinafes larger fhan B' , say ti and tj. 

Given tt, we can form vr' from tt by preserving fhe images equal fo 1, replacing fhe images equal fo 2 
by j, and replacing fhe images equal fo 3 by n. This has fhe effecl of replacing every insfance of ti by 
fi + • • • + fj-i and every insfance of f 2 by + • • • + fn-i in fhe upshiff and downshiff inequalities from 
fhe proof of Theorem 12.151 

Hence, because fhe solutions for vr include all poinfs wifh ti > B' and t 2 > B', fhe solufions for tt' 
include all poinfs 

+ ■ ■ ■ + fi + • • • + tj—i > B' 

and 

fj + • • • + fn—1 > B' . 

Therefore, t' is a feasible poinf for vr'. 

Thus, S'” satisfies Lemma [T5l so is periodic. □ 
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